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We study phase diagrams of the Hubbard model on anisotropic triangular lattices, which also 
represents a model for K-type BEDT-TTF compounds. In contrast with mean-field predictions, 
path-integral renormalization group calculations show a universal presence of nonmagnetic insula- 
tor sandwitched by antiferromagnetic insulator and paramagnetic metals. The nonmagnetic phase 
does not show a simple translational symmetry breakings such as flux phases, implying a gen- 
uine Mott insulator. We discuss possible relevance on the nonmagnetic insulating phase found in 
At-(ET) 2 Cu 2 (CN) 3 . 



When the kinetic and Coulomb repulsion energies are 
severely competing, the ground states of many-body elec- 
tron systems can be highly nontrivial. The Mott tran- 
sition provides a typical example of such nontrivial be- 
havior. If the electronic orbitals constituting the band 
near the Fermi level form geometrically frustrated lattice 
structure, the ground state can be even more nontrivial, 
because the spin entropy is not easily released by simple 
symmetry breaking such as the antiferromagnetic (AF) 
order, and the possiblity of some exotic quantum liq- 
uid state opens. Theoretically, regular triangular lattice 
has been proposed as such a typical structure Q, while 
the nature of the ground state of the Hubbard model on 
this lattice still remains a challenge because of the lack 
of theoretical tools. Reliability of mean field approaches 
becomes questionable under strong quantum fluctuations 
while the numerical studies have been limited to small 
clusters because of the lack of efficient algorithm. 

In this letter, we obtain the ground-state phase di- 
agram of the Hubbard model on anisotropic triangular 
lattice called t-t'-TH model by using recently developed 
path-integral renormalization group (PIRG) method 
with systematic inclusion of quantum fluctuations. In 
the t-t'-TH model, when one of the transfers (<') on a 
unit triangle becomes different from the other transfers 
t as in Fig. [j], the model continuously connects the right 
triangular lattice (t'/t = 1) to the regular square one 
it' /t — 0). We study this general model at half filling. 

In particular, the case between t'/t ~ 0.5 and 1.0 has 
attracted a particular interest because tight binding fit of 
the band structure by the extended Hiickel approxima- 
tion for K-type BEDT-TTF compound suggests that this 
region may provide a minimal model after eliminating 




FIG. 1: Lattice structure of tt'-TH model on triangular lattice 
with anisotropic transfers t and t' . 



the internal degrees of freedom in the dimerized two ET 
molecules || . In terms of clarifying the metal-insulator 
(MI) transition and the accompanied superconductivity, 
the ground state of this model has been a subject of in- 
tensive studies but mainly on the Hartree-Fock or FLEX 
levels [@, [|, @|- Because significant correlation and 
quantum effects are expected as described above, it is 
desired to clarify the phase diagram of the basic t-t'-TH 
models itself by fully taking account of quantum fluctu- 
ations. 

Our result shows remarkably a generic emergence of 
nonmagnetic insulating state sandwitched by a Mott MI 
transition and an AF transition in contrast with the pre- 
vious mean-field predictions, while a nonmagnetic insu- 
lating state is also suggested in different Hubbard models 
on the square lattice with next-nearest neighbor trans- 
fers 0. We also find that the nonmagnetic insulators do 
not show simple symmetry breakings such as dimer, pla- 
quette, and flux phases, in support for the appearance of 
genuine Mott insulator distinct from band insulators. 

The genuine Mott insulating state not adiabatically 
connected to the band insulator has been a subject of 
long-standing search since Anderson's proposal |Q . This 
proposed RVB state is in basic sense realized in the 
ID systems , while it remains a challenge for multi- 
dimensional systems. We stress that such RVB state may 
be found more easily near the bandwidth-control metal 
to Mott insulator transition, where quantum mechani- 
cal fluctuations of spin as well as charge destroy trivial 
symmetry breakings and the local moment is relatively 
small. In the light of our results, we argue that this cate- 
gory of genuine Mott insulator may indeed be realized in 
k-(ET)2Cu2(CN)3, which seems to be nonmagnetic and 
insulating at ambient pressure down to 1.4K. 

We introduce t-t'-TH model on a triangular lattice 
shown in Fig. |l| with the Hubbard Hamiltonian H = 

- £<;,,•), CT t (4 c jV + H.c.) +E (fe ,o,a * ( c L c l° + H - c -) + 
U^2^ =1 n^riii w ith the standard notation. In this letter, 
we apply the recently developed PIRG method Q to clar- 
ify the ground state in the plane of U/t and t'/t. This al- 
gorithm allows us to start from and improve a mean-field 
Hartree-Fock solution and reach the ground-state by tak- 
ing account quantum fluctuations in a systematic fashion. 
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FIG. 3: Momentum distribution on 14 x 14 lattice at t'/t 
0.5 for U/t = 3.8 and 4.2. 



FIG. 2: Averaged double occupancy D as a function of U/t 
for /;'/£ = 0.5(filled circle), 0.8(open square) and l.O(asterisk). 
The plotted error bars are only from the size extrapolation. 
The errors produced by the variance extrapolation [2] in PIRG 
is not shown and are at most 0.02. 



By increasing the dimension of truncated Hilbert space 
in a nonorthogonal basis numerically optimized by the 
path-integral operation, a variance extrapolation is taken 
to reach the true ground state of finite size systems in a 
controlled way. We took number of Slater basis func- 
tions up to around 300. This numerical method does not 
have difficulties such as the negative sign problem in the 
quantum Monte Carlo calculations. In the present study, 
we take system size up to N = 196 sites (14 x 14) with 
the periodic boundary condition. Using the results at 
various sizes, we extrapolate to the thermodynamic limit 
using appropriate scaling procedure and eliminate finite 
size effects. 

To see the MI transition, we calculate the singularity 
of the ground-state energy E g as a function of the con- 
trol parameter, U/t. The level crossing with a first-order 
transition or singularity with a continuous character is 
detected as a signature of the MI transition with comple- 
mentary analysis of the charge gap and the momentum 
distribution. The AF transition is studied by equal-time 
spin correlation functions in the momentum space. We 
also calculate several density wave correlations to dis- 
cuss possible symmetry breakings such as staggered flux 
as well as dimer correlations. 

In Fig. we show dE g /dU as a function of U. Jumps 
of the averaged double occupancy D = (n^riii) = 
dEg/dU /AT +1/4 signal first-order MI transitions induced 
by level crossings at U/t = 4.1 ± 0.1, and 5.0 ± 0.2 for 
choices t'/t — 0.5, and 0.8, respectively, while a contin- 
uous transition is plausible at 5.2 ± 0.2 for t'/t = 1.0. 
These transitions show similar character to MI transi- 
tions identified by charge gap formation and qualitative 
change of the momentum distributions for other types 
of Hubbard models ||. Fig. || exemplifies that the mo- 
mentum distributions of the i-i'-TH model indeed show 
transition from a typical metallic behavior to that of an 
insulator across this boundary . 
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FIG. 4: Size scaling for the peak value of the equal-time spin 
structure factor S(q pea k) at t'/t = 0.5. 



The magnetic transition is probed by the Fourier trans- 
form of the equal-time spin correlation, S{q) defined by 

S{q) = wEfj (Si ■ S j )e u *< R '- R i), where S t is the 
spin of the i-th site with Ri representing the coordinate 
of the i-th site. At t'/t = 0.5 the spin correlation shows 
sharp commensurate paek at (tt, it) for U/t > 5.7 while it 
remains rather small and incommensurate for U/t < 5.6. 
The size scaling in Fig. |] leads to U dependence of the 
staggered moment m — ^limpf^oo S(q pea k)/N as in 
Fig. pL where a first-order (or very sharp continuous) AF 
transition is identified at U/t — 5.65 ± 0.05. A small 
jump of dEg/dU seen in Fig. || at this point also sup- 
ports the first-order character. For larger values of t'/t, 
at t'/t = 0.8 and 1.0, the size scaling of the spin correla- 
tion shows the absence of magnetic symmetry breaking 
at all momenta and all U studied, namely for U/t < 10. 

The phase diagram of the t-t'-TH model constructed 
from our calculations is illustrated in Fig. ^. A remark- 
able feature is the emergence of nonmagnetic insulat- 
ing phase sandwitched by MI and AF transitions. This 
is in contrast with the Hartree-Fock results where the 
MI transition occurs at larger U/t than the AF tran- 
sition. The triangular lattices were also studied in the 
spin-1/2 Heisenberg model corresponding to the limit 
U — » oo ||, [l(]] . Although the decisive conclusion cannot 
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FIG. 5: Staggered magnetization in the thermodynamic limit 
as a function of U/t at t/t = 0.5. 



FIG. 7: Size scaling for the d density wave (stuggered flux) 
correlation A d = \D ld {q peak )\/N for t'/t = 1.0. 
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FIG. 6: Phase diagram of t-tl TH model in the plane of U/t at 
t'/t. AFI,NMI and PM represent AF insulator, nonmagnetic 
insulator, and paramagnetic metal, respectively. Calculations 
have been done at the cross points. The arrow shows specu- 
lated boundary from AFI to the dimer insulator in the limit 
U — > oo for the J1-J2 model JlQ ], 

be drawn due to the lack of powerful techniques, it was 
claimed from a small cluster studies or series expansions 
that either long-range AF order or dimer order appear in 
this model on a regular triangular lattice. This suggests a 
possibility that the reduction of the local moment due to 
charge fluctuations at U reduced to a finite value drives 
the destruction of the order. In fact, for example, the 
local moment mi = y/ (n) — 2D/2 is reduced to mi ~ 0.4 
at the phase boundary U/t = 5.2 at t'/t = 1.0. 

Insulators with some translational symmetry breakings 
such as the AF and dimer orders may be adiabatically 
continued to band insulators due to the band gap for- 
mation through extension of the unit cell and the folded 
Brillouin zone, and hence do not belong to the genuine 
Mott insulator discussed above. We first have to note 
that it is hard to prove the existence of the genuine Mott 
insulator because it is difficult to prove the absence of 
all the possible translational symmetry breakings. Un- 



der this circumstance, an available way to get insight on 
this problem is to examine every symmetry breaking ever 
proposed and judge the plausibility of such a phase. 

Along this line of motivation, we have further calcu- 
lated correlations for several symmetry breakings claimed 
in the literature such as dimer orde r [|l0"| , [HJ , plaquette or- 
der Jl2]| and various density waves []l3| in this remarkable 
nonmagnetic insulating phase. Here the density wave 
correlation is defined by D a (q) = ^A^(q)A Q (q)) where 

A Uq) = Efe, CT /"( fc ) c L c fc+<7,<r- We have studied the 
cases fi s (k) = cosk x + cosk y , f2 S {k) — 2cosk x cosk yi 
fid(k) — cosk x —cosky, and f2d(k) = 2 sin k x sin k y . We 
note the case Id represents the order parameter for the 
staggered flux (or d-density wave) state |ll| . Fig. |^ shows 
a typical size scalings of these density wave correlations 
for the t-t'-TH model. Other types show similar behav- 
iors. It does not seem to show symmetry breakings and 
the correlation actually remains rather short ranged in 
all the parameter space we have studied (0 < U/t < 10 
and 0.5 < t'/t < 1.0). The possibility of dimer, plaquette 
and density wave orders have also been examined in the 
Hubbard model on a square lattice with diagonally cross- 
ing transfers between next-nearest neighbor sites. The 
size scalings again suggest the absence of these transla- 
tional symmetry breakings |l~4|]. Although these results 
do not exclude a possiblity for the translational symme- 
try breakings other than those studied here, we conclude 
that the theoretically proposed phases so far are unlikely 
to be realized in our phase hence the existence of genuine 
Mott insulating state becomes much more plausible. 

Our result leads us to an intriguing view for recent 
experimental results on organic compounds. As we dis- 
cussed above, the t-t'-TH model provides a simplest ef- 
fective Hamiltonian for the K-type ET compounds if we 
may take the dimerization of two parallel ET molecules 
large j|. The HOMO band is split by the dimerization 
and the upper band provides the effective Hubbard model 
at half filling. The effective interaction for U in this case 

is taken as U eff = 2t bl + 2f^[l - ^1 + where 
Uet is the original onsite Coulomb repulsion for the ET 
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molecule and tbi is the intradimer transfer. 

Among available compounds, the band structure calcu- 
lation 15, 1(J suggests k-(ET) 2 Cu 2 (CN) 3 has the largest 
ratio of t'/t ~ 1.0 (t ~ 0.055 eV and t' ~ 0.058 eV 
) with rather large U e ff/t ~ 8.2. This compound is 
close to the metal-insulator phase boundary because a 
small pressure around 0.05GPa drives the insulating to 
the superconducting phase. The ESR measurement of k- 
(ET) 2 Cu2(CN) 3 at ambient pressure as well as the sus- 
ceptibility seems to suggest that the insulating phase re- 
mains nonmagnetic with broad structure of the spin sus- 
ceptibility IQ. Very recent NMR data |l8| also suggest 
that the 1/Ti remains constant above 100K while it starts 
substantially decreasing below 50K and there exists no 
signature of the magnetic transition down to 1.4K so far. 
No indication of lattice distortion has been reported. 

It is very attractive to relate these experimental re- 
sults to our calculated phase diagram with the insulat- 
ing phase without symmetry breakings, because the pa- 
rameter values deduced from the band structure calcula- 
tion correspond to t'/t = 1.0 where we obtain large non- 
magnetic insulating phase. In other compounds such as 
deuterated K-(ET) 2 Cu[N(CN) 2 ]Br, the first-order tran- 
sition between a superconductor and an AF insulator is 
observed. These compounds have relatively small param- 
eters t'/t ~ 0.5-0.6 (if). In our result at t'/t = 0.5, there 
still exists nonmagnetic insulating phase. This discrep- 
ancy is speculated to be due to an oversimplification to 
the effective single-band model. In fact, internal degrees 
of freedom of dimer stabilizes metals more than insulator, 
while the geometrical frustration is relaxed by an assy- 
metric distribution of electrons on a dimer and the AF 
order may not be suppressed strongly, which results in 
shrink of the nonmagnetic insulator phase. Necessity of 
multi-band models was also pointed out in different con- 
texts JH], pCfl . Quantitative analysis of this multi-band 
effect is left for further studies. If the AF and MI transi- 
tions become close, lattice distortion accompanied by the 
MI transition necessarily enhance the first-order nature 
also leading to the shrink of the nonmagnetic insulator. 



The absence of strong AF correlations near the MI 
transition implies that the mechanism of the supercon- 
ductivity seen in k-(ET) 2 Cu 2 (CN)3 under pressure may 
not be directly connected with AF fluctuations. In our 
PIRG calculation, although the pairing correlations defi- 
nitely remain short-ranged for U/t = 3 at t'/t — 0.5 and 
for U/t = 4 at t'/t = 0.8 and 1.0, the d-wave pairing 
correlations near the MI boundary does not show a good 
convergence which indicates the necessity of larger num- 
ber of Slater determinants in PIRG for this quantity to 
get converged results, implying large quantum fluctua- 
tions. This will be discussed elsewhere. 

We also point out, however, a fundamentally interest- 
ing region still waits for experimental realization: In ET 
dimerized compounds, stronger dimerization is required. 
More sophisticated realization of nonmagnetic insulator 
and metals near it in the frustrated lattice would be de- 
sired also in other materials. 

In summary, we have studied the Hubbard model 
on anisotropic triangular lattice (i-t'-TH model). The 
ground state is obtained by the PIRG method, where 
the Hartree-Fock results are systematically improved to 
reach the true ground state. The phase boundary of 
bandwidth control MI and AF transitions in the pa- 
rameter space of t'/t and U/t shows that a nonmag- 
netic insulator appears between these two transitions. 
Simple translational symmetry breakings such as flux 
phases remain short ranged and genuine Mott insulating 
state seems to be realized. We have pointed out that 
k-(ET) 2 Cu 2 (CN)3 may belong to this category of the 
genuine Mott insulator within the available data. The 
degeneracy of the ground state as well as the spin ex- 
citation gap in the nonmagnetic insulator phase is an 
intriguing future problem to be clarified. 
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